Closing the loop between neural network simulators and the OpenAI Gym

Since the enormous breakthroughs in machine learning over the last decade, functional neural network models are of growing interest for many researchers in the field of computational neuroscience. One major branch of research is concerned with biologically plausible implementations of reinforcement learning, with a variety of different models developed over the recent years. However, most studies in this area are conducted with custom simulation scripts and manually implemented tasks. This makes it hard for other researchers to reproduce and build upon previous work and nearly impossible to compare the performance of different learning architectures. In this work, we present a novel approach to solve this problem, connecting benchmark tools from the field of machine learning and state-of-the-art neural network simulators from computational neuroscience. This toolchain enables researchers in both fields to make use of well-tested high-performance simulation software supporting biologically plausible neuron, synapse and network models and allows them to evaluate and compare their approach on the basis of standardized environments of varying complexity. We demonstrate the functionality of the toolchain by implementing a neuronal actor-critic architecture for reinforcement learning in the NEST simulator and successfully training it on two different environments from the OpenAI Gym

Closed loop interactions between spiking neural network and robotic simulators based on MUSIC and ROS

In order to properly assess the function and computational properties of simulated neural systems, it is necessary to account for the nature of the stimuli that drive the system. However, providing stimuli that are rich and yet both reproducible and amenable to experimental manipulations is technically challenging, and even more so if a closed-loop scenario is required. In this work, we present a novel approach to solve this problem, connecting robotics and neural network simulators. We implement a middleware solution that bridges the Robotic Operating System (ROS) to the Multi-Simulator Coordinator (MUSIC). This enables any robotic and neural simulators that implement the corresponding interfaces to be efficiently coupled, allowing real-time performance for a wide range of configurations. This work extends the toolset available for researchers in both neurorobotics and computational neuroscience, and creates the opportunity to perform closed-loop experiments of arbitrary complexity to address questions in multiple areas, including embodiment, agency, and reinforcement learning

25th annual computational neuroscience meeting: CNS-2016

The same neuron may play different functional roles in the neural circuits to which it belongs. For example, neurons in the Tritonia pedal ganglia may participate in variable phases of the swim motor rhythms [1]. While such neuronal functional variability is likely to play a major role the delivery of the functionality of neural systems, it is difficult to study it in most nervous systems. We work on the pyloric rhythm network of the crustacean stomatogastric ganglion (STG) [2]. Typically network models of the STG treat neurons of the same functional type as a single model neuron (e.g. PD neurons), assuming the same conductance parameters for these neurons and implying their synchronous firing [3, 4]. However, simultaneous recording of PD neurons shows differences between the timings of spikes of these neurons. This may indicate functional variability of these neurons. Here we modelled separately the two PD neurons of the STG in a multi-neuron model of the pyloric network. Our neuron models comply with known correlations between conductance parameters of ionic currents. Our results reproduce the experimental finding of increasing spike time distance between spikes originating from the two model PD neurons during their synchronised burst phase. The PD neuron with the larger calcium conductance generates its spikes before the other PD neuron. Larger potassium conductance values in the follower neuron imply longer delays between spikes, see Fig. 17.Neuromodulators change the conductance parameters of neurons and maintain the ratios of these parameters [5]. Our results show that such changes may shift the individual contribution of two PD neurons to the PD-phase of the pyloric rhythm altering their functionality within this rhythm. Our work paves the way towards an accessible experimental and computational framework for the analysis of the mechanisms and impact of functional variability of neurons within the neural circuits to which they belong

Dynamics, Plasticity and Learning in a Neural Network Model of the Basal Ganglia

Abstract Organisms are able to learn from reward and punishment to cope with unknown situations, in psychology this phenomena is called reinforcement learning. Experimental studies provide strong evidence that the mechanisms behind reinforcement learning in the mammal brain are located in the basal ganglia, an subcortical area known to be involved in decision making and the planning of movement [squire2013fundamental, purves2004Neuro]. The neurotransmitter dopamine plays a key role in reinforcement learning [schultz1997neural]. The adaption of behavior is believed to be driven by dopaminergic plasticity in the striatum, the input nucleus of the basal ganglia. In order to investigate the neural mechanisms behind reinforcement learning, an existing spiking neural network model of the basal ganglia [jitsev2012learning2] is improved and extended. The mathematical and structural basis of this model is provided by temporal difference learning, a reinforcement learning algorithm known from the field of machine learning [barto1998reinforcement]. Abstract While being an effective learning algorithm, though, temporal difference learning is not motivated from a biological perspective. Understanding how learning from reward and punishment takes place in the mammal brain requires biological constraints. This thesis addresses this point by implementing homeostatic regulation mechanisms and investigating neural circuits which implement value computation and action selection in the striatum. To improve the biological plausibility of the model, the synaptic plasticity rules of the dopaminergic feedback in the model are re-calculated and simplified. Finally, the plausibility of winner-takes-all circuits in the dorsal striatum by strong lateral connection between fast spiking interneurons is discussed. Abstract This work establishes a basis for future research on the origin of abnormal dopaminergic plasticity leading to dysfunctional states of the basal ganglia like those observed in diseases like Parkinson or Tourette syndrome

NEST: Simulator for large-scale neural networks

NEST (The Neural Simulation Tool, www.nest-simulator.org, github.com/nest/nest-simulator) is an open-source simulator for spiking neural network models that focuses on the dynamics of large structured networks rather than on the exact morphology of individual neurons. The development of NEST is coordinated by the NEST initiative (www.nest-initative.org). NEST is ideal for networks of spiking neurons of any size, for example models of information processing e.g. in the visual or auditory cortex of mammals, models of network activity dynamics, e.g. laminar cortical networks or balanced random networks and models of learning and plasticity. In this 120-minute tutorial, a general introduction to the NEST simulator will be given and demonstrations on how to create and execute NEST models using the PyNEST module in Python will be provided, covering basic and more advanced examples of usage